Efficient Variational Inference for Gaussian Process Regression Networks

In multi-output regression applications the correlations between the response variables may vary with the input space and can be highly non-linear. Gaussian process regression networks (GPRNs) are flexible and effective models to represent such complex adaptive output dependencies. However, inference in GPRNs is intractable. In this paper we propose two efficient variational inference methods for GPRNs. The first method, gprn-mf, adopts a mean-field approach with full Gaussians over the GPRN’s parameters as its factorizing distributions. The second method, gprn-npv, uses a nonparametric variational inference approach. We derive analytical forms for the evidence lower bound on both methods, which we use to learn the variational parameters and the hyperparameters of the GPRN model. We obtain closed-form updates for the parameters of gprn-mf and show that, while having relatively complex approximate posterior distributions, our approximate methods require the estimation of O(N) variational parameters rather than O(N) for the parameters’ covariances. Our experiments on real data sets show that gprn-npv may give a better approximation to the posterior distribution compared to gprn-mf, in terms of both predictive performance and stability.